Program Documentation CAITEST Computer Adaptive Test Engine Thomas B. Wiggers 528 Cliffview Drive Brandon, Ms. 39042 Program Setup Required Programs: CAITEST.EXE Main Program COMUTL.DOC Documentation for Print Screen Disable & Enable NOPSCR.COM Disables SHIFT+PRTSCR PSCR.COM Enables SHIFT+PRTSCR STARTCAI.BAT Main Program Loader WRITECFG.EXE Creates Configuration File DataFile Naming Conventions: "Filename".INI Configuration files for the CAITEST engine "QFIL"+"LD"+"N"+".MCH" LOW difficulty question file where "N" is the file number "QFIL"+"N"+".MCH" MODERATE difficulty question file where "N" is the file number "QFIL"+"HD"+"N"+".MCH" HIGH difficulty question file where "N" is the file number "QFIL"+"LD"+"N"+".SFL" LOW difficulty solution file where "N" is the file number "QFIL"+"N"+".SFL" MODERATE difficulty solution file where "N" is the file number "QFIL"+"HD"+"N"+".SFL" HIGH difficulty solution file where "N" is the file number SCORFIL.DAT Score File The name of the associated LOG file is also listed. "Filename".LOG LOG files These files provide a complete record of student performance on a given exam. File Sets The data files for the CAITEST program may be set up in either of two ways. When a competency-based examination is selected, each file set must consist of 3 files. "QFIL"+"LD"+"N"+".MCH" LOW difficulty question file where "N" is the file number "QFIL"+"N"+".MCH" MODERATE difficulty question file where "N" is the file number "QFIL"+"HD"+"N"+".MCH" HIGH difficulty question file where "N" is the file number Questions are read from these files according to the criteria established by the WRITECFG program, and described below. If a solution file is to be presented, there must be a solution file for EACH level of question file. The solution files are to be named as follows: "QFIL"+"LD"+"N"+".SFL" LOW difficulty solution file where "N" is the file number "QFIL"+"N"+".SFL" MODERATE difficulty solution file where "N" is the file number "QFIL"+"HD"+"N"+".SFL" HIGH difficulty solution file where "N" is the file number When a percent correct examination is selected, only one question file is necessary in a file set. It should be named as follows: "QFIL"+"N"+".MCH" where "N" is the file number If solution files are selected, there MUST be a solution file for each numbered question file. The solution file should be named as follows: "QFIL"+"N"+".SFL" where "N" is the file number The CAITEST program does not contain a question entry, edit, or print functions. It was designed to work with question files created by the TEST GENERATOR program, and all entry and editing of questions MUST be done with that program. Follow each step exactly as explained below to set up the CAITEST for operation: 1. Copy the program files either to a blank, formatted diskette, or to a subdirectory on your hard drive. 2. Change to the diskette or subdirectory by typing "A:" or "B:" followed by pressing the "ENTER" key, or by the "CD" command to change directory. 3. Once the directory containing the programs has been made the current directory, type "WRITECFG" and press the "ENTER" key. The Configuration Program: "WRITECFG.EXE" Purpose: Provides an easy way to configure the CAITEST program by asking the configuration questions and then creating the initialization file (.INI suffix files). This file is an ASCII text file, and may be edited with any text editor, once the format is understood. Enter the Subject Area or Topic for the Examination Enter here what you wish to use for the examination title. There is a maximum length allowed, and you will be prompted if you exceed that length, and be allowed to reenter the title. Enter Examination Access Code: Determines the filename of the initialization file. Must be a legal filename of no longer than 8 characters. The student will be prompted to enter this access code when they begin the testing program, so that the proper initialization of the program may be accomplished. The initialization filename will be the ACCESS CODE + ".INI", and will be located in the directory from which "WRITECFG" is run. Do You Wish a Difficulty-based Exam: A "Y" here indicates that grading of the exam and selection of questions should be based on student competency. A "N" indicates that question selection should be from medium-difficulty files only, and that grade be determined as a percentage of correct answers. Enter the Number of Questions to be randomly selected.. A "N" answer to the question above indicates that the examination is to be scored by simple percentage scoring, not competency based. Only one question is necessary per Qfile in this case, however a better approach is to create a single Qfile containing all the questions for the examination, say 50. Answering "50" to this question means that all 50 questions will be selected and presented to the student for the exam. A "40" here means that 40 of the 50 would be randomly pulled. The program automatically calculates the total length of the examination. For example, 3 file sets (Qfiles), where 10 questions are to be selected randomly from each file, gives a total test length of 30 questions. How many (maximum) high difficulty questions... Enter here the maximum number of high difficulty questions the student will be asked if they are successful with questions of moderate difficulty. If the number of questions in a given file is less than this number, all questions in the file will be selected and the program will continue. How many (maximum) medium difficulty questions... Enter here the maximum number of medium difficulty questions the student will be asked if they are unsuccessful with questions of high difficulty OR when a new file set is begun. If the number of questions in a given file is less than this number, all questions in the file will be selected and the program will continue. How many (maximum) low difficulty questions... Enter here the maximum number of low difficulty questions the student will be asked if they are unsuccessful with questions of medium difficulty. If the number of questions in a given file is less than this number, all questions in the file will be selected and the program will continue. _________ NOTE: If any of the numbers entered above are LESS than the total number of questions in a file, the total number of questions asked from a file set will be adjusted to that number. For example: Suppose you have entered "3" as the numbers of HIGH, MEDIUM, and LOW difficulty questions to be selected. There are 7 questions in QFILLD1.MCH, 22 questions in QFIL1.MCH, and 2 questions in QFILHD1.MCH. The total number of questions which will be selected from file set #1 is adjusted downward to "2". Since you have indicated that you want to ask 3 medium difficulty questions, the student will receive 2 medium difficulty questions, and then move to the next file. Always make sure that there are AT LEAST the number of questions in the question files that you specify in the above settings to prevent this adjustment from occurring. _________ What will be the minimum acceptable competency index for successful completion of the exam? See "Program Logic & Scoring" section at the end of this documentation for an explanation of C.I. determination. After how many questions do you want to move to the next file.. See "Program Logic & Scoring" section 8 at the end of this documentation for an explanation of this option. Do you want a solution... Solution files are ASCII text files that provide a general indication to the student as to how the questions in a file should be solved. For example, if a file contains molarity questions, the solution file might contain an explanation of the formulas required for solving the problems. A solution file is only relevant when all questions within a file are related. If solution files are selected, ALL files must have solution files, even if they simply state that there is no solution file necessary for a particular group of questions. Solution files must be named according to the naming conventions stated at the beginning of this documentation, and must reside in the same directory as the datafiles . Should the Student be given immediate feedback... If the answer to this question is "Y" then students will be told whether or not the answer they have selected is correct or incorrect before the program moves on to the next question. If the student's answer is incorrect, they will also be told which was the correct response. If no entry is made here, the program defaults to NO FEEDBACK (N). Enter the MAXIMUM TIME.. At the end of the allotted time the student is presented with a message that they have exceeded the allotted time, and that they have been unsuccessful in their attempt at the examination. Enter the PATH or location of the data files: Type here the EXACT PATHNAME where the datafiles for a given examination reside. They do not have to be in the same directory as the CAITEST programs, although it is acceptable for them to be. If the PATH is a subdirectory, a backslash "\" MUST be included in the pathname as the last character (see examples). If the PATH is the Root Directory, the "\" is not necessary. Example pathnames: A: A:\CAITEST\ C: C:\CAITEST\LABMATH\ PROGRAM COLORS Allows the user to customize the colors of the program. They do not have to be entered or altered. Press 'ENTER' to accept all the default colors. To Adjust Color: Colors #2 and #3 of the colors list are used to adjust background and foreground color, respectively. Enter the appropriate number from the following list: 1 Dark Blue 9 Light Blue 2 Dark Green 10 Light Green 3 Dark Cyan 11 Light Cyan 4 Dark Red 12 Light Red 5 Purple 13 Light Purple 6 Brown 14 Light Yellow 7 White 15 Bright White 8 Gray The user is next presented with a confirmation screen. If everything is as needed, type "Y" to store the values to the configuration file. 4. Type "STARTCAI" and press 'ENTER' to begin the CAITEST program. 5. When prompted for the Access Code, enter the code exactly as specified in the WRITECFG program above. The Access Code is NOT case-sensitive. 6. The remaining activities are prompted on-screen and should be self-explanatory. Program Logic & Scoring 1. Low Difficulty questions: 1 point. Medium Difficulty questions: 2 points High Difficulty questions: 3 points 2. Competency Index Calculations: Sum of Points Received for Correct Answers as Shown Above C.I.= ----------------------------------------------------------------------------------- Total Questions Answered example: A student has answered 2 low, 3 medium, and 1 high difficulty question correctly out of 15 questions presented. Point values assigned are as follows: 2 low at 1 point each = 2 points 3 med at 2 points each= 6 points 1 high at 3 points each= 3 points ------- 11 points total 11 C.I. = ------ or .733 15 3. A student is presented initially with medium difficulty questions. After the student answers all questions, the computer will evaluate the C.I. of the student. If the CI is above the minimum required, the computer will then present high difficulty questions to the student from the same file set. If after evaluation the CI is found to be below the minimum required, the computer will present low difficulty questions. 4. If, after attempting the high difficulty questions, the CI is found to be below the acceptable minimum, the student is presented with more medium difficulty questions from the same file set. 5. If, after attempting the high difficulty questions the CI is above the acceptable minimum, the computer will move on to the next file set. 6. If the student has attempted low difficulty questions, and the CI is below the minimum, the computer will move to the next file set and select questions of medium difficulty. 7. If the student answers all of the low difficulty questions which are presented, the computer will present additional medium difficulty questions, same file. 8. If a student has successfully completed the medium difficulty questions, been presented with high difficulty questions and after the high difficulty questions has a competency index below the minimum acceptable, the student will receive more moderate level questions. It is conceivable that the student might miss one or more of these moderate questions. If, after receiving the moderate level questions, the CI is below the minimum acceptable, the program will present low difficulty questions for the student to attempt. This could continue indefinitely, except for the value for the "Maximum Number of Questions per File Set" which has been entered by the instructor. When the number of questions asked the student from a single file set is equal to this number, the program moves to the next file set, regardless of the calculated CI. 9. Questions that are skipped may be returned to at the end of the examination. When a question is skipped, the CI is calculated on the basis of the questions actually answered. There is one possible "worst-case" scenario: If all questions on the entire examination are skipped, the CI for each file is calculated to be "0". Therefore, the computer will present only medium and low difficulty questions. If the minimum competency level set by the instructor is high enough, it might be possible that the student would not be able to pass the examination by answering all questions on the second pass. example: 20 questions are presented. All are skipped on the first pass. This will result in the presentation of 10 medium difficulty and 10 low difficulty questions. (2+2 per file). The result of the calculation of the CI, assuming all questions were answered correctly on the second pass would be: 10 X 1 point = 10 points 10 X 2 points= 20 points ---- 30 points 30 C.I.= ------- or 1.5 20 If the instructor has stipulated 1.8 as the minimum acceptable CI, then the student will not receive a passing score. 10. When the examination is completed, the student receives information concerning the number of questions which were missed, a list of the files that were attempted, and the CI for each individual file set. There is also a cumulative CI which represents the calculated CI for the entire examination. 11. The student scorefile contains the following information: Time of Student Log-on Student Name Name of Examination Log-File Number of Questions on Examination Percent Score on Examination Cumulative CI for Entire Examination, if competency-based Length of Time Required to Complete Examination CI for Each File Set Attempted (in order, left to right), if competency-based CI for Questions That Were Skipped on the First Pass, if competency-based 12. The Examination Log-File contains the following information: TOTQST Total Number of Questions for Examination FILE Filename of Questions Presented to the Student QUES Question Number in File S Student Response (* indicates that the student skipped the question) A Actual Answer Tracking Values for Several Scoring Variables for use in troubleshooting: FS Score for current Qfile TS Total Score for all questions presented COR Number of Correct Responses on Examination % Percent Score on Examination CI CI for File Set MQ Maximum Number of questions per file -ACTUAL allowed QMX Maximum Number of questions per file -REQUESTED by configuration program Computer Adaptive Test Program Version 2.0 Upgrade Revisions 02/17/94 Revions to the version 2.0 test driver include the following: 1. The program now allows the selection of multiple questions from a single QFILE, when utilizing a non competency-based examination. For example a 50 question test may be generated from a single 50 question file, instead of 50 QFILES with a single question each. The program tests each question selected to determine if it has been used previously, and will not select the same question again within a single examination. 2. The program allows the instructor to select whether or not students will receive immediate feedback when they miss a question on the examination, i.e. "No the correct answer should have been 'B'" ,or whether students will not be notified immediately when they miss a question. 3. Students may see the time remaining for the exam, or the percent grade through the prior question on the exam (when % based) by selecting the appropriate menu option. 4. Student logfile output is now formatted and tabular. 5. Instead of being presented with only a listing of missed questions at the end of the examination, students are allowed to review the questions they missed. The questions are presented again and the correct answer is shown. 6. For 286 and higher processors, the following programs are provided, in which the compiled code is optimized for these machines. These programs execute MUCH faster on newer machines: CAIT286.EXE CAITEST.EXE CATUT286.EXE CATUTIL.EXE TGEN286.EXE TESTGEN.EXE WRCFG286.EXE WRITECFG.EXE 7. Default colors have been optimized for both color and monochrome monitors. Default selections will work equally well on both types of monitors. Any problems or questions should be addressed to: Thomas Wiggers 528 Cliffview Drive, Brandon, Ms. 39042 tel: day (601)984-6309 night (601) 992-1373 collect. The programs carry a lifetime guarantee, and are NOT copy protected. They MAY NOT be transferred or given to another party. Since the programs are being routinely updated and improved, any constructive suggestions will be appreciated. caidoc.wp5 February 21, 1994